The vanishing discount approach in Markov chains with risk-sensitive criteria

In this paper stochastic dynamic systems are studied, modeled by a countable state space Markov cost/reward chain, satisfying a Lyapunov-type stability condition. For an infinite planning horizon, risk-sensitive (exponential) discounted and average cost criteria are considered. The main contribution is the development of a vanishing discount approach to relate the discounted criterion problem with the average criterion one, as the discount factor increases to one, i.e., no discounting. In comparison to the well-established risk-neutral case, our results are novel and reveal several fundamental and surprising differences. For example, the limit of the (normalized) discounted costs, as the discount vanishes, is not equal to the average cost, but rather equal to an arithmetic mean of the average cost over a range of values for the risk sensitivity coefficient. Other significant contributions made in this paper are the use of convex analytic arguments to obtain appropriately convergent sequences and a verification theorem for the case of unbounded solutions to the average cost Poisson equation arising in the risk-sensitive case. Also of importance is the fact that our developments are very much self-contained and employ only basic probabilistic and analysis principles.